SPMI: Use diffed bytes for jit-analyze and print more info #61254
Conversation
Fix the misleading total bytes displayed when diffing with SPMI. If one
of the JITs encountered missing data, the code bytes would not be
included, while the other JIT could still have it included if it did not
encounter missing data.
Also add more information about missing SPMI data/successful replays
printed. For replays that is useful to be able to gauge whether there is
still ok coverage after a large change. For diffs, we print a warning
for missing SPMI data that the diff summary may be misleading.
Example for a successful replay:
Clean SuperPMI replay (219868 contexts processed)
Example for a replay with missing data:
SuperPMI encountered missing data for 6 out of 27272 contexts
Example warning printed:
Warning: SuperPMI encountered missing data during the diff. The diff
summary printed above may be misleading.
Missing with base JIT: 0. Missing with diff JIT: 6. Total contexts:
27272.
dotnet-issue-labeler
bot
added
the
area-CodeGen-coreclr
|
Tagging subscribers to this area: @JulieLeeMSFT Issue DetailsFix the misleading total bytes displayed when diffing with SPMI. If one Also add more information about missing SPMI data/successful replays Example for a successful replay: Example for a replay with missing data: Example warning printed:
|
|
@sandreenko for your PR this prints the following to console now: Summary of Code Size diffs:
(Lower is better)
Total bytes of base: 7170863 (overridden on cmd)
Total bytes of diff: 7170866 (overridden on cmd)
Total bytes of delta: 3 (0.00 % of base)
diff is a regression.
relative diff is a regression.
Top file regressions (bytes):
3 : 11295.dasm (2.86% of base)
1 total files with Code Size differences (0 improved, 1 regressed), 1 unchanged.
Top method regressions (bytes):
3 ( 2.86% of base) : 11295.dasm - System.TimeSpan:Negate():System.TimeSpan:this
Top method regressions (percentages):
3 ( 2.86% of base) : 11295.dasm - System.TimeSpan:Negate():System.TimeSpan:this
1 total methods with Code Size differences (0 improved, 1 regressed), 1 unchanged.
--------------------------------------------------------------------------------
Warning: SuperPMI encountered missing data during the diff. The diff summary printed above may be misleading.
Missing with base JIT: 0. Missing with diff JIT: 6. Total contexts: 27272. |
|
cc @dotnet/jit-contrib |
|
@jakobbotsch It looks like superpmi.py will use the new "override base/diff metric" logic if the |
|
I agree that sounds like a fine way to do it. FWIW, my envisioning was to eventually pass the base and diff summary metrics files themselves (instead of just the numbers of the code size metric) to jit-analyze and have jit-analyze use directly numbers from there to present other metrics as well (e.g. perf score once we have a good way of getting it (#52877), memory allocations). |
ghost
locked as resolved and limited conversation to collaborators
Fix the misleading total bytes displayed when diffing with SPMI. If one
of the JITs encountered missing data, the code bytes would not be
included, while the other JIT could still have it included if it did not
encounter missing data.
Also add more information about missing SPMI data/successful replays
printed. For replays that is useful to be able to gauge whether there is
still ok coverage after a large change. For diffs, we print a warning
for missing SPMI data that the diff summary may be misleading.
Example for a successful replay:
Clean SuperPMI replay (219868 contexts processed)
Example for a replay with missing data:
SuperPMI encountered missing data for 6 out of 27272 contexts
Example warning printed:
Warning: SuperPMI encountered missing data during the diff. The diff summary printed above may be misleading.
Missing with base JIT: 0. Missing with diff JIT: 6. Total contexts: 27272.